Java爬虫某购物平台商品评论 |
您所在的位置：网站首页 › 京东签到脚本auto 苹果手机 › Java爬虫某购物平台商品评论 |
Java爬虫某购物平台商品评论
|
准备条件
浏览器 + HttpClient4 jar包 + fastjson 爬取过程 1. 获取商品评论区真实url打开京东 - 选择商品 - 点击商品评价 - 单击f12打开开发者工具 - network - 工具栏搜索“product”
复制真实url
单击 preview
我们发现，这不是标准的json格式，它多了个“fetchJSON_comment98(” 和 “);”,待会获取到json文本后我们需要对字符串把多余的部分去掉，可以用正则表达式，或者对直接substring（）截取 掐头去尾 正则表达式标准化JSON json = json.replaceAll("\\w+\\(",""); json = json.replaceAll("\\);{1}",""); 完整代码 import com.Jsoup.HttpUtil3; import com.alibaba.fastjson.JSONArray; import com.alibaba.fastjson.JSONObject; import org.apache.http.clienthods.CloseableHttpResponse; import org.apache.http.clienthods.HttpGet; import org.apache.http.impl.client.CloseableHttpClient; import org.apache.http.impl.client.HttpClients; import org.apache.http.util.EntityUtils; import java.io.IOException; import java.util.regex.Pattern; public class Crawler { public static void main(String[] args) throws InterruptedException { String url = "https://club.jd.com/comment/productPageComments.action?callback=fetchJSON_comment98&productId=12598703&score=0&sortType=5&page=0&pageSize=10&isShadowSku=0&fold=1"; CloseableHttpClient httpClient = HttpClients.createDefault(); HttpGet httpGet = new HttpGet(url); httpGet.setHeader("User-Agent","Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/110.0"); CloseableHttpResponse response = null; try { response=httpClient.execute(httpGet); String json = EntityUtils.toString(response.getEntity()); //掐头去尾 json = json.replaceAll("\\w+\\(",""); json = json.replaceAll("\\);{1}",""); JSONObject object = JSONObject.parseObject(json); JSONArray rawComment = JSONArray.parseArray(String.valueOf(object.get("comments"))); //遍历输出 rawComment.forEach(item->{ System.out.println(((JSONObject)item).get("content")); }); } catch (IOException e) { e.printStackTrace(); } } } 运行结果
|
【本文地址】